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Josephine: 

METHOD AND SYSTEM FOR AUTOMATIC MEASUREMENT AND TRACKING OF LOGS, INDUSTRIAL 
WOOD AND BOARDS 



Description 
Field of invention 

[0001] This invention relates to measurement and traddng of objects by 
use of photogrammetric methods. The invention targets the situation where 
bundles of such objects are handled for being collected in larger bimdles or 
for further processing of individual objects. During this action the individual 
objects of a small bundle will be visible to a large extend. In its simplicity 
the idea then is by photogrammetric methodologies to obtain information 
of a particular object from imagery of one or more small bundles where it 
appears. More specifically the three dimensional (3D) extend of an object 
and properties related to its surface such at textme and other significant 
characteristics are extracted. Capturing imagery while grabbing and releas- 
ing the small bxmdles and/or at successive points during the handling gives 
the information necessary to keep track of the transportation path in time 
and space of each individual object. Thus the methodology in total provides 
measurements and tracking of the individual objects. 

[0002] Clearly the invention encompasses a wide range of possible applicar- 
tions. A generic example is an industrial process where a conmiodity travels 
through several steps of subprocesses during the manufacturing. The speci- 
fication here is focused on a unique design for use in the forestry and wood 
processing industry. 

[0003] During the last decades forestry has undergone a substantial degree 
of mechanization. Today wood is harvested and processed at an increasing 
speed and the wood industries makes up large units. Further the madiinery 
used applies highly developed technologies. As a standard felling machinery 
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and saws at sawmills are equipped with various computer technologies to 

aid software for optimization in each step of the procurement process. At 
so the same time there is a growing public awareness of the management of 

natural resources and the utilization of .those resources that are harvested. 

It is generally agreed that future advances relating to wood processing are 

to be obtained from further optimizations of the wood procinement process. 

The present invention facilitates optimizations throughout wood procure- 
ss ment process by providing detailed information of the wood (objects) at any 

stage where the system is applied. 

Greneral section 

[0004] The present invention is a real-time computer vision and tracking 
system to automatically locate and measure the size and quality of individual 
60 hard and softwood logs, pieces of industrial wood, and boards, hereinafter 
logs. 

[0005] This system is to rationalize, and increase efficiency and measurement 
accuracy throughout the wood prociirement process. Further, the precise 
tracking of the logs allows for accurate docmnentation in e.g., pursuit of 

65 wood certification. 

[0006] By a computer vision system is understood a single or a series of 
computer sensor sj^ems with integrated GPS, The vision system locates and 
measures the logs at several stages of some wood procurement process. The 
results of one or more of these measurements axe to be used independently 

70 or in conjunction to give precise and unbiased estimates of position, size and 
quality of logs at any specific stage. 

[0007] By tracking is understood a record of the positions of a recognized log 
along the procurement process where the system is applied e.g., the positions 
of a specific log from felling to any stage along the procurement process where 
75 the system is no longer applied. 
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[0008] The system will satisfy diffiarent immediate demaads for mfoimation 
of various actors along the procurement process. In particulax: i) buyers 
and sellers demand for accurately measured and classified trading units and 
documentation hereof, and ii) the need to back-track logs and other wood 

80 commodities to their origin in the forest for certification purposes. 

[0009] Further, the information coillected by the system up to any given 
stage (operation) of the procurement process can be utilized for optimization 
purposes in subsequent stages of the procurement process. This increases the 
efficiency of the procurement process. , 

85 [0010] A schematic representation of the wood prociu^ment process is shown 
in Figure 1. The action pattern of logs being loaded and unloaded by similar 
devices e.g., cranes, is repeated at all operations throughout the procurement 
process. Therefore the system is designed to apply sizxiilar computer sensor 
systems and similar image analysis algorithms at all stages. 

go [0011] Typically logs are traded at one or more of the operations listed 
in Figure 1. Thus the unit for trade is a varying collection of logs and the 
system is designed to compute aggregate values for such collections of logs. 
[0012] The system can be applied at any action or set of actions e.g., an 
operation listed in Figure 1. In consequence it is the responsibility of buyers 

95 and sellers to decide, on what stages it is appropriate to apply the system. 
Back-tracking boards to their origin in the forest does naturally require the 
S3^em being applied at most unload and load points (actions). 
[0013] The present invention suggests measuring individual logs using a 
stereo vision system integrated with GPS. The GPS system provides the lo- 

100 cation in space of the equipment that handles the logs and the vision system. 
Combined with automatic recognition of the logs in imagery captured by 
the vision system the path of each individual log can be mapped to provide 
tracking of the logs. 

[0014] The core idea of the vision system is that it captures imagery of 
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105 the log3 during load/unload actions. During these actions only a. few logs 
are handled at a time by a crane or similar device. Imagery q£ a bundle 
of few logs allows for a complete recognition of each individual log in the 
btmdle. Candidate mount points of vision system af e e.g., grab on harvester 
or forwarder, body of harvester or forwarder, truck at plant, conveyer belt 

uo or any other place where the logs can be seen from vision system. 

[0015] Further, imagery of the stacks (collections of logs) that axe being 
loaded or imloaded are captinred continuously to monitor exactly where each 
bimdle of logs are placed or taken from. In this way stacks are considered ' 
cohorts of individual logs with their individual properties such as transporta- 

m tion path and size. Thus aggregate quantities and distributional statistics at 
the stack level can be computed to characterize a stack. 
[0016] The idea of measuring logs on the fly by remote sensing and stacks 
being cohorts of logs with an associated record of information at the log level 
is an advance compared to todays practices. Common practices is to measure 

120 stacks in a separate process and characterize stacks by their outer measure 
and other aggregate quantities. 

[0017] Each individual log is being monitored several times during the 
procurement line. At any load/unload action stereo imagery of eadi log while 
located in the unload stack, the crane and the load stack is being captured. 

125 This comprehensive information source forms the basis of an image analysis 
algorithm to estimate the size and location of the logs with a high level of 
accuracy. Integrating imagery across several operations in Figure 1 in the 
image analysis adds to the level of accuracy that can be obtained. 
[0018] The image analysis algorithm is designed to adopt information from 

130 many sources. Hence the algorithm allows data fusion from sources such 
as field inventory, harvester, regional prior information on size and quality, 
and information dynamically gathered during a forest operation to ensure 
self-calibration. 
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[0019] An important by-product of the real-time implementation of the 
system is that the influence of decay and oth^ biological factors can be 
quantified. Especially if there is a time-lag between the creation of a stack 
and it bemg picked up, using the system at both of these operations makes 
it possible to monitor changes in the stack. 

[0020] The present invention is considered applicable for other industrial 
uses where 3D objects are being handled in a isimilar repeated load/unload 
fashion in stacks. 

Specific section 

[0021] The system is designed to be build from standard components that 
complies with industry standards to ensure portability and low cost. In 
particular standard cameras, standard image formats, standard graphics hard 
and software, standard computer units with ax^companying operating systems 
suffice to build an operatioiial system. 

[0022] One example of an actual implementation of the this system is based 
on CCD digital cameras with a spatial resolution of 2-5 mm^ per pixel for 
objects located at 2-15 m distance in physical space from the vision system. 
The image format is coinpressed tiff and the software implementation of this 
system written in C/C++/C# runs a windows or Linux box. The graphics 
procedures runs throtigh an accelerated 3D graphics card using the OpenGL 
language. 

[0023] In the following is by machinery understood some ma diinery that 

handles logs along the wood procurement process. The device niaed by such 

macbinery to handle the logs will be rrferred to as a grab. 

[0024] Some examples of machinery are Vahnet and Ponsse harvesters 

and forwarders, Mack and Volvo trucks with trailers build to load logs and 

equipped with jibs, and genuine cranes at the saw or paper mills. 

[0025] The stereo vision system is to be mounted on the maxiiinery such 
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that a continuous series of images of the logp that are being handled can be 
captured. The ideal strata is to capture one or more set of images of the 
load and unload stacks and of the bundle of logs in the grab for each bundle 

165 of logs being handled. 

[0026] As an example harvesters have lights mounted pairwise on their jibs 
arms and shielded by a frame. A set of cameras are conveniently mounted 
next to the lights within these frames. When the operator fells a tree and at 
each time a log is cut off the vision system takes images. For a forwarder the 

no cameras are mounted similarly, but here the images are taken whenever a new • 
cohort of logs i grabbed or released and at regular time intervals in-between 
or whenever the orientation of the cohort in the grab is optimal relative to 
the cameras. 

[0027] The GPS unit is mounted on the machinery too. Rrom its continuous 
175 logging of signals the absolute orientation and location in space of the ma- 
chinery is known. Given the orientation of the grab relative to the machinery 
the location in space of the grab is computed. 

[0028] Logs are three-dimensional (3D) solid objects. In consequence the 
image analysis algorithms are implemented by use of software routines sup- 
180 ported by standard 3D graphics hardware. Computing images of a virtual 
3D universe that represents the real world stacks and bundles of 3D logs then 
is fast and strai^t forward. 

[0029] The resulting information about stacks or bundle of logs computed 
from the images are subsequently stored in a central database. This informa- 
las tion include basicly estimates on size, location and quality of the individual 
logs. Further, aggregate values at stack level as well as image scenes (movie 
picttnres) of part of or of the whole recorded operation shotild be stored in 
the database too. 

[0030] The central database can be located on any pre-selected machinery 
2^0 or office computer. The only requirement is . that the other computers are 
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able to acscess the central computer by an online connection e.g., the Internet 
by a phone card or a similar standard device. This way the system is robust 
against hardware failure on any link in the procurement line. 
[0031] The image analysis algorithm for tracking and size estimation is 
conceptually split into two stages. Stage 1 is a coarse recognition of the 
logs taken from or added to a stack, and the logs held in the grab. Stage 

1 is accomplished by template matching or other filtermg and provides ap- 
proximate location and size of logs m the virtual 3D world. Stage 2 is to 
infer accurately about each individual log from the image data available and 
other prior information. The approximate configuration of the virtual 3D 
world from Stage 1 is used as initial value for stage 2. Stage 2 is inore in- 
volved and applies a formal statistical analysis. The outcome of the Stage 

2 are configurations of the virtual 3D world (one for each captured image 
nsed m the analysis) that describes the real world best given the information 
available. In other words, Stage 2 provides the transportation paths of the 
logs and their size and quality. 

[0032] Both of Stage 1 and Stage 2 require a specification of the virtual 
3D world that mimics the real world. That is, a mathematical model is. set 
up to describe the 3D shape of the logs and their transportation path. This 
model is then adjusted (estunated) to fit best possible with the actual taten 
images. 

[0033] A possible mathematical model for a log is that the stem center fol- 
lows a 2nd, 3rd or 4th order polynomial (in a plane) and that cross-sections 
perpendicular to the stem center are circular with diameter a linear func- 
tion of distance to the stem base (Figure 4). Alternatively diameter can be. 
specified by stem taper functions commonly described in literature. 
[0034] Given a specification of the virtual world the mapping into image 
space (image formation) is required to compute the images of the virtual 
world. It is by comparison of these computed images and the actual taken 
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220 images that the mathematical model can be estimated. 

[0035] As an example if the vision system is based o^ CCD cameras the 
mapping from object space (the physical world) into image space is the com- 
position of a central projection and the deterioration by the lens and irregu- 
larities induced by the chip. The two latter together makes up the so-called 

225 imier orientation and is a specific property of each camera. The imier orienta- 
tion is established separately from a test scene in a laboratory. The location 
and attitude of the cameras in the vision system relative to one-another to- 
gether with the model for image formation makes up a stereo vision system. 
That is, a system that allows for 3D reconstruction of the objects of interest 

230 (the logs). 

[0036] The Stage 1 filter that singles out individual logs in the grab operates 
on a complex source of information including one or more of: grab 'Svidth", 
location of grab in space (implicitly orientation of logs), exact record of the 
logs handled if taken from stack aheady monitored by the system, other prior 

235 information on log size eg., from harvester. 

[0037] The Stage 1 filter that singles out individual logs taken or added 
to a stack operates on successive images of the stadc to detect changes in 
its surface. Candidate locations of logs are identified from these changes 
(Figure 3). Information about the location of the grab where it took or 

240 added logs and the btmdle that was handled is used in support of this filter. 
[0038] In essence the Stage 1 filters map the complete transportation path 
of each mdividual log and the tracking is completed. Note that image data 
and GPS data are used in conjunction to complete this task. 
[0039] The inference framework for Stage 2 is Bayesian where the like- 

245 lihood term /(J; 0) is the density of the image data / under a statistical 
model parameterized by 0 and 7r{0) is the prior on 0. The statistical model 
encompasses the model of the virtual 3D universe, the image formation pro- 
cess and the randomness of noise in the imagery. The posterior p is givai 
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by pie-, I) oc ir{e)f{I\e). Since 9 inchides the paxameterization of the vir- 
^0 tual 3D world, maxtauzing p provides mi estimate of the configuration of the 
3D real-world logs. A natural choice of the prior it is knowledge about the 
size distribution of the logs under study, lypically this information is eaaly 
obtained from growth models or yield tables. 

[0040] Ifedmically each log in the virtual world is represented by a discrete 
255 spatial object during the maidmization of the posterior. The set of pomts 
that spaas the log are derived from the underlying mathematical model. The 
triangulation between these points then makes up the surface o.f the log. Let 
ff be the parameter that parameterizes a particular log. Then 6* is a subset 
of e. Having maximized p with respect to e therefore ptovides the best value 
2M for ff. Volume and curvature and other important properties of the log are 
therefore best computed from the value olff. 

[0041] Since logs are solid objects with a certain regulaxiiy in shape, the 
parameter 0* can not be considered a completely free parameter. Thus the 
posterior p is maximized under suitable smoothness constraints on the shape 

205 of logs. Further different logs axe in principle not allowed to occupy the same 
physical space. In practice this constraint is relaxed a bit to accommodate 
for the fact that is not a perfect representation of the real-world logs. 
[0042] If the system.isappUed at several stages along the wood procurement 
line, the image data J is the aggregate image data from all the consecutive 

270 images captured. This impUes that the 3D configuration of each log is esti- 
mated from image data captured from different dfrections rdative to the log 
Tyi inimigim^ g the ocduded volume of the logs. 

[0043] The system operates in absolute imits in the sense that it produces 
the measured sizes in meters or some other absolute unit. Technically this 
»s requires the scale be known in the imagery captured by the stereo vision 
system. In recognition of the fact that the system operates in a disturbed 
environment, marks on the grab or other parts of the machinery is used in 
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addition to a known base line of the vision ^rstem to obtain scale.- 
[0044] The Stage 2 image analyds ^ves a coarse assessment of wood quaiity 
280 e.g., based on curvahuie of a log and the appeaianoe of growth rings at cross 
sectional cuts of logs. Additional image analysis routines are appUed to 
measure rot percent, knots, baxk tejdanre changes to get as much information 

about wood quality as possible. 

[0045] When a stack of logs is completed the system computes aggregate 
«5 quantities for the stack. The set of quantities computed may be changed 
dynamicaUy by the user. Typical aggregate quantities are: number of logs • 
in the stack, total volume of the logs in the stack, and size and quality 
distribution of the logs in the stack. 

[0046] Whenever 1<^ are collected on some naachinery and transported 

»o to some other location for unload the real-time requirement on the image 
analysis algorithms can be relaxed. It is not until the unload action takes 
place that the system must provide the image analysis results. This fact 
is particularly useful in the first stage where the system is appUed is on a 
forwarder in the forest. At the first stage no prior information about the log 

sgs is available and successful log recognition and size estimation may require 
an extended computation time. Usage of the system at subsequent stages 
along the procurement process will require less computer run time because 
detsdled prior information about the logs are available. 
[0047] The vision ^rstem is mounted on the machinery so that it can capture 

aw images of both the grab and the load and unload stacks. The ideal situation 
is that the system be mounted such that for each bimdle of logs that is being 
handled images of both grab and stacks can be captured. Some candidate 
mount points are jib arm or the body of the maxiiinery as shown in Figure 2. 
[0048] A unique feature of the system is that the measuring device, i.e. the 

395 vision system, is not in contact with the logs. This unpUes that the system is 
less prone to deterioration, and thus the demand for continuous calibration 
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of the system while heing APPlied is low. This is an advantage to measturing 
devices that are mounted in the felling device of a harvester. , 
[0049] The present invention suggests both the triadring and the size estima- 
tion tasks sohred by the system. As time progresses and e.g. bio-informatics 
technologies evolve, the tracking task may be completed from the DNA foot- 
print of each log or some other recognition procedure. By virtue the system 
is designed to incorporate such information in the Stage 2 image analysis. In 
other words, the system is designed to be complemented by other information 
sources that aid the tracking and size estimation. 

[0050] The most critical factors to the system are the sensor conditions and 
the number of logs that the grab handles. Operating the system during e.g., 
night time with a sensor sensible to visible light thus requires the logs being 
lighted by some artificial light soiiroe. If a particular log is occluded by other 
logs when located in the stack or in the grab, the system can only provide an 
approximate estimate of its size. The solution space for the voliune occupied 
by that particular log is however quite small so that the overall performance 
of the system i not adversely effected. 

[0051] If tiae order of logs m a stack is modified by some outside factor or 
its geometric constitution has changed much, the change is recognized by the 
system and it starts monitoring if logs are missing in the stack. 
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